GyroFlow+: Gyroscope-Guided Unsupervised Deep Homography and Optical Flow Learning

Existing homography and optical flow methods are erroneous in challenging scenes, such as fog, rain, night, and snow because the basic assumptions such as brightness and gradient constancy are broken. To address this issue, we present an unsupervised learning approach that fuses gyroscope into homography and optical flow learning. Specifically, we first convert gyroscope readings into motion fields named gyro field. Second, we design a self-guided fusion module (SGF) to fuse the background motion extracted from the gyro field with the optical flow and guide the network to focus on motion details. Meanwhile, we propose a homography decoder module (HD) to combine gyro field and intermediate results of SGF to produce the homography. To the best of our knowledge, this is the first deep learning framework that fuses gyroscope data and image content for both deep homography and optical flow learning. To validate our method, we propose a new dataset that covers regular and challenging scenes. Experiments show that our method outperforms the state-of-the-art methods in both regular and challenging scenes.Comment: 12 pages. arXiv admin note: substantial text overlap with arXiv:2103.1372

T-PickSeer: Visual Analysis of Taxi Pick-up Point Selection Behavior

Taxi drivers often take much time to navigate the streets to look for passengers, which leads to high vacancy rates and wasted resources. Empty taxi cruising remains a big concern for taxi companies. Analyzing the pick-up point selection behavior can solve this problem effectively, providing suggestions for taxi management and dispatch. Many studies have been devoted to analyzing and recommending hot-spot regions of pick-up points, which can make it easier for drivers to pick up passengers. However, the selection of pick-up points is complex and affected by multiple factors, such as convenience and traffic management. Most existing approaches cannot produce satisfactory results in real-world applications because of the changing travel demands and the lack of interpretability. In this paper, we introduce a visual analytics system, T-PickSeer, for taxi company analysts to better explore and understand the pick-up point selection behavior of passengers. We explore massive taxi GPS data and employ an overview-to-detail approach to enable effective analysis of pick-up point selection. Our system provides coordinated views to compare different regularities and characteristics in different regions. Also, our system assists in identifying potential pick-up points and checking the performance of each pick-up point. Three case studies based on a real-world dataset and interviews with experts have demonstrated the effectiveness of our system.Comment: 10 pages, 10 figures; The 10th China Visualization and Visual Analytics Conferenc

Single-Image-Based Deep Learning for Segmentation of Early Esophageal Cancer Lesions

Accurate segmentation of lesions is crucial for diagnosis and treatment of early esophageal cancer (EEC). However, neither traditional nor deep learning-based methods up to today can meet the clinical requirements, with the mean Dice score - the most important metric in medical image analysis - hardly exceeding 0.75. In this paper, we present a novel deep learning approach for segmenting EEC lesions. Our approach stands out for its uniqueness, as it relies solely on a single image coming from one patient, forming the so-called "You-Only-Have-One" (YOHO) framework. On one hand, this "one-image-one-network" learning ensures complete patient privacy as it does not use any images from other patients as the training data. On the other hand, it avoids nearly all generalization-related problems since each trained network is applied only to the input image itself. In particular, we can push the training to "over-fitting" as much as possible to increase the segmentation accuracy. Our technical details include an interaction with clinical physicians to utilize their expertise, a geometry-based rendering of a single lesion image to generate the training set (the \emph{biggest} novelty), and an edge-enhanced UNet. We have evaluated YOHO over an EEC data-set created by ourselves and achieved a mean Dice score of 0.888, which represents a significant advance toward clinical applications

Supervised Homography Learning with Realistic Dataset Generation

In this paper, we propose an iterative framework, which consists of two phases: a generation phase and a training phase, to generate realistic training data and yield a supervised homography network. In the generation phase, given an unlabeled image pair, we utilize the pre-estimated dominant plane masks and homography of the pair, along with another sampled homography that serves as ground truth to generate a new labeled training pair with realistic motion. In the training phase, the generated data is used to train the supervised homography network, in which the training data is refined via a content consistency module and a quality assessment module. Once an iteration is finished, the trained network is used in the next data generation phase to update the pre-estimated homography. Through such an iterative strategy, the quality of the dataset and the performance of the network can be gradually and simultaneously improved. Experimental results show that our method achieves state-of-the-art performance and existing supervised methods can be also improved based on the generated dataset. Code and dataset are available at https://github.com/megvii-research/RealSH.Comment: Accepted by ICCV 202

EmoCo: Visual analysis of emotion coherence in presentation videos

Emotions play a key role in human communication and public presentations. Human emotions are usually expressed through multiple modalities. Therefore, exploring multimodal emotions and their coherence is of great value for understanding emotional expressions in presentations and improving presentation skills. However, manually watching and studying presentation videos is often tedious and time-consuming. There is a lack of tool support to help conduct an efficient and in-depth multi-level analysis. Thus, in this paper, we introduce EmoCo, an interactive visual analytics system to facilitate efficient analysis of emotion coherence across facial, text, and audio modalities in presentation videos. Our visualization system features a channel coherence view and a sentence clustering view that together enable users to obtain a quick overview of emotion coherence and its temporal evolution. In addition, a detail view and word view enable detailed exploration and comparison from the sentence level and word level, respectively. We thoroughly evaluate the proposed system and visualization techniques through two usage scenarios based on TED Talk videos and interviews with two domain experts. The results demonstrate the effectiveness of our system in gaining insights into emotion coherence in presentations.Comment: 11 pages, 8 figures. Accepted by IEEE VAST 201

Characterization of protein-protein interactions between the nucleocapsid protein and membrane protein of the avian infectious bronchitis virus

Avian infectious bronchitis virus (IBV) is one of the major viral respiratory diseases of chickens. Better understanding of the molecular mechanism of viral pathogenesis may contribute significantly to the development of prophylactic, therapeutic and diagnostic reagents as well as help in infection control. Avian IBV belongs to the Coronaviridaes and is similar to the other known coronaviruses. Previous studies have indicated that protein–protein interactions between nucleocapsid (N) and the membrane (M) proteins in coronavirus are related to coronavirus viral assembly. However, cases of IBV are seldom reported. In this study, yeast two-hybrid and  co-immunoprecipitation techniques were applied to investigate possible interactions between IBV N and M proteins. We found that interaction of the N and M proteins took place in vivo and the residues 168 – 225 of the M protein and the residues 150 - 210 of the N protein were determined to be involved in their interaction. These results may provide some useful information on the molecular mechanism of IBV’s N and M proteins, which will facilitate therapeutic strategies aiming at the disruption of the association between membrane and nucleocapsid proteins and indicate a new drug target for IBV.Key words: Co-immunoprecipitation, membrane protein, nucleocapsid protein, protein-protein interaction, yeast two-hybrid

Amplifying the Music Listening Experience through Song Comments on Music Streaming Platforms

Music streaming services are increasingly popular among younger generations who seek social experiences through personal expression and sharing of subjective feelings in comments. However, such emotional aspects are often ignored by current platforms, which affects the listeners' ability to find music that triggers specific personal feelings. To address this gap, this study proposes a novel approach that leverages deep learning methods to capture contextual keywords, sentiments, and induced mechanisms from song comments. The study augments a current music app with two features, including the presentation of tags that best represent song comments and a novel map metaphor that reorganizes song comments based on chronological order, content, and sentiment. The effectiveness of the proposed approach is validated through a usage scenario and a user study that demonstrate its capability to improve the user experience of exploring songs and browsing comments of interest. This study contributes to the advancement of music streaming services by providing a more personalized and emotionally rich music experience for younger generations.Comment: In the Proceedings of ChinaVis 202

VoiceCoach: Interactive evidence-based training for voice modulation skills in public speaking

The modulation of voice properties, such as pitch, volume, and speed, is crucial for delivering a successful public speech. However, it is challenging to master different voice modulation skills. Though many guidelines are available, they are often not practical enough to be applied in different public speaking situations, especially for novice speakers. We present VoiceCoach, an interactive evidence-based approach to facilitate the effective training of voice modulation skills. Specifically, we have analyzed the voice modulation skills from 2623 high-quality speeches (i.e., TED Talks) and use them as the benchmark dataset. Given a voice input, VoiceCoach automatically recommends good voice modulation examples from the dataset based on the similarity of both sentence structures and voice modulation skills. Immediate and quantitative visual feedback is provided to guide further improvement. The expert interviews and the user study provide support for the effectiveness and usability of VoiceCoach.Comment: Accepted by CHI '2

Improved performance and stability of perovskite solar modules by interface modulating with graphene oxide crosslinked CsPbBr3quantum dots

Perovskite solar cells (PSCs) are one of the most prominent photovoltaic technologies. However, PSCs still encounter great challenges of scaling up from laboratorial cells to industrial modules without serious performance loss while maintaining excellent long-term stability, owing to the resistive losses and extra instability factors that scale quadratically with the device area. Here, we manifest a concept of multifunctional interface modulation for highly efficient and stable perovskite solar modules (PSMs). The advisably designed multifunctional interface modulator GO/QD crosslinks the CsPbBr3 perovskite quantum dots (QDs) on the conductive graphene oxide (GO) surfaces, which significantly improve charge transport and energy band alignment at the perovskite/hole transporting layer interface to reduce the charge transport resistance while passivating the surface defects of the perovskite to inhibit carrier recombination resistive losses. Moreover, the GO/QD interlayer acts as a robust permeation barrier that modulates the undesirable interfacial ion and moisture diffusion. Consequently, we adopt a scalable vacuum flash-assisted solution processing (VASP) method to achieve a certified stabilized power output efficiency of 17.85% (lab-measured champion efficiency of 18.55%) for the mini-modules. The encapsulated PSMs achieve over 90% of their initial efficiency after continuous operation under 1 sun illumination and the damp heat test at 85 °C, respectively. This journal isThe authors acknowledge financial from the National Natural Science Foundation of China (21875081, 91733301, and 51972251), the Chinese National 1000-Talent-Plan program, the Foundation of State Key Laboratory of Coal Conversion (Grant No. J18-19-913), and the Frontier Project of the Application Foundation of Wuhan Science and Technology Plan Project (2020010601012202)